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ABSTRACT 

This paper res ponds to c alls for furthe r research on learning effectiveness of instructional technologies by comparing 
learning'and 'attitudes' across three instruction al environments (multimedia, textbookand in-class instruction), and 
proposing a quasi-experimental research design approach. The paper reports the results of pilot studies conducted 
with the aim of refining the research framework, assessing the experimental conditions, and obtaining preliminary 
results. This research framework and the pilot studies are described . 



INTRODUCTION 

Since technology is so widespread in society, different 
stakeholders believe that educational strategies should 
reflect this expanding phenomenon by introducing 
instructional technology in the classroom. They often 
assume that the effectiveness of technology has been 
proven once and for all (Roblyer et al, 1997). Instead, 
the learning effectiveness of instructional technology is 
yet to be proven. In spite of the several studies (Nugent, 
1982; Mayer & Anderson,! 991) and meta-analyses 
(Kulik, Kulik & Shwalb, 1986; Roblyer, Castine & 
King, 1988; Kulik, 1994; Khalili & Shashaani, 1994;) 
competing to answer the question: “do computers and 
other related technologies make a difference in 
learning? the question on the comparative performance 
of instructional technology is notyet answered. Several 
scholars are calling for further research (Liao, 1998; 
Jones & Paolucci, 1998; Fletcher-Flinn & Gravatt, 
1995). 



The authors try to answer the call for further research 
with a quasi-experimental design which compares 
'learning' and 'attitudes' (satisfaction and appeal) in 
subjects gaining knowledge of project management 
topics by either attending a class presentation, reading a 
textbook, or using an interactive multimedia CD-ROM. 
The subjects ofthe study are graduate and undergraduate 
students enrolled in business administration degree 
programs at a major East-coast University. The study is 
replicated, under comparable conditions, with the two 
groups. The purpose of the replication is to explore 
whether learning effectiveness and satisfaction vary by 
age (graduate and undergraduate), and by subject topic 
characteristics. 

Because the results from the implementation of the 
quasi-experiment are not available at the present time, 
this paper focuses in describing the theoretical 
framework, the research design, and the results of the 
pilot studies that precede the actual testing. The results 
from the analysis will be available in December 2000. 
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THE LEARNING EFFECTIVENESS OF 
INTERACTIVE MULTIMEDIA SYSTEMS 

The authors’ expectations based on the literature review 
is that the instructional conditions that best support 
multiple representation modes have a higher impact on 
learning specific content. Instructional technologies such 
as interactive multimedia allow the integration of 
multiple media in a specific delivery environment, 
therefore promoting multiple representations of 
knowledge. The structural features (organization of 
content, modes of delivery) of interactive multimedia 
have an impact in the construction of mental models 
(Jonassen, 1990) and, particularly, a positive effect on 
the cognitive system. 

Tergan (1997) states that the cognitive processing of 
multiple external representations (such as the ones in 
interactive multimedia and hypermedia) matches 
principles of encoding variability and encoding 
specificity (Tulving, 1983). The encoding specificity 
principle requires that learning situations contain 
recognizable clues thathelp learnersunderstand, locate, 
and recall information; and that realistic contexts (like 
the ones in interactive multimedia systems) generally 
result in better performance. The principle of encoding 
variability is also supported by interactive mltimedia 
multiple representation modes. The fact that information 
is encoded in a variety of formats increases the chances 
that at least one ofthe representations corresponds to the 
context of use in a particular situation (Hammond, 
1993). 

The value of interactive multimedia learning is also 
supported by the principles of cognitive flexibility theory 
(Jacobson & Spiro, 1995); multimodal cognitive 
processing (Cunninhgham et al., 1993); and indi- 
vidualized and self-regulated instruction (Duffy & 
Knuth, 1990). According to the cognitive flexibility 
theory, interactive multimedia environments employ 
multiple ways to represent knowledge in instructional 
activities, thus reflecting the nature of complex 
knowledge. This contributes to the development of 
cognitive structures that are flexible enough to be 
applicable to different contexts, enhancing knowledge 
transfer. Paivio ( 1 986) finds that using visuals and other 
representational modes entails dual coding, which 
enables humans to better retain information. Paivio ’s 
theory is supported by recent developments of 
multimedia approaches to learning (Engelkamp & 



Zimmer, 1994). Remembering, for example, may be 
enhanced because information is represented in different 
cognitive systems. 

The learning effectiveness argument for multimedia is 
also supported by the multiple intelligences theory 
(^Gardner, 1984, \ 993) dindemotional intelligence theory 
(Goleman, 1995). Multiple symbolic modes promote 
synergistic interactions among several intelligences 
(spatial, kinesthetic, logical-mathematical, musical, 
linguistic, interpersonal, intrapersonal) and benefit 
learning in the various domains - cognitive, affective, 
psychomotor - (Kemp, 1994). Computer-based media 
support the social dimensions of human learning, 
interpersonal and intrapersonal intelligences, to an 
astonishing degree (McLellan, 1996). 

By affecting different domains (cognitive, affective and 
psychomotor) multimedia also allows mass- 
customization of instruction, since instruction offered in 
several modes is adaptable to the individual learner. 
Different learners are more attuned to different types of 
representations (Keller, 1983) and implementing 
reinforcing activities may enable them to “explicitly or 
“implicitly selectthe activity that is most “customized 
to theircogniti ve skills (McLellan, 1 996). In addition, by 
shifting between different representations, learners get 
multiple practice (M arks-Tarlow, 1995). 

RESEARCH DESIGN 

In this framework, the main question ofthe researchers 
is 'Whether interactive multimedia applications are 
effectivelearning toolscompared to traditionallearning 
environments (textbook, or in-class instruction).' To test 
the learningeffectiveness of interactive multimedia, this 
study adopts the definition of learning in Merrill’s 
performance-contentmatrix model (1983). Based on this 
definition, the independent variables* (the three 
instructional settings, such as the class presentation, the 
textbook, and the multimedia CD-ROM) effects on the 
learning of project management topics {dependent 
variable) are examined through comparing subjects’ 
recall and application performance in specific content 
areas (recall and application of facts, concepts, 
principles, and procedures). Subjects’ attitudes 
{dependent variable) across the instructional settings are 
investigated through learners’ comparative satisfaction 
with the instructional materials (see Figure 1). 
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FIGURE 1 

QUASI-EXPERIMENTAL MODEL FOR TESTING THE 
COMPARATIVE EFFECTIVENESS OF INSTRUCTIONAL TECHNOLOGIES 




The intervening variables include: student 

characteristics such as gender, age, prior knowledge, 
computer abilities, and learningpreferences; and, subject 
characteristics, such as complexity of the topic (high 
and low complexity). Assessing learners’ prior 
knowledge of the topic and reporting their demographic 
breakdown reinforces the research design. It enables 
comparing and contrasting the treatment and control 
groups, as well as developing a better understanding of 
the characteristics of the students surveyed. 

Because the effectiveness of the instructional materials 
may also depend on learners’ prior experience in 
management, the study utilizes a pre-test & posttest 
design to assess learners prior knowledge of the topic. In 
addition, the demographic survey collects data on 
learners prior work experience (see, for example, F/gwre 
2), major, and computer utilization. 

Related questions associated to the model presented in 
Figure I and based on Merrill's (1983) performance- 
content matrix include: 



Which environment is more effective at achieving 
learning objectives (recall or application)? 

Which environment is more effective at delivering 
a specific type of topic (with high/low complexity)? 

Which environment, if any, is more appealing for 
the learners’ groups (graduate, undergraduate) 
targeted? 

The responses to these questions based on the 
implementation of the study will be available in 
December 2000. Although it is expected that it will be 
difficult to make clear-cut and distinct statements on the 
instructional effectiveness of technology (interactive 
multimedia may be more effective for specific content 
areas, such as recall of facts and principles, but less 
effective in applications of procedures), the authors set 
up a study that tries to overcome the limitations 
recognized in earlier studies. 
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DIFFICULTIES AND REMEDIES 

There are many factors that can influence the results of 
effectiveness studies. The researchers recognize the 
difficulty and address the objections brought to similar 
comparative learning research. In particular, Clark 
(1983) offered many objections to the value of the 
analyses that claimed comparatively positive outcomes 
in educational technologies. According to Clark (1983) 
the most common sources of confounding evidence in 
earlier studies appeared to be the uncontrolled effects of: 
a) instructional method or content differences between 
treatments that are compared; and b) novelty effect for 
newer media, which tends to disappear over time. 

1. The research design ofthis study addresses Clark’s 
considerations and tries to isolate learning from the 
medium by limiting the instructional time, 
controllingthe learning environment, and assessing 
the role of intervening variables. 

2. To address Clark’s first concern (point a), the 
researchers focus on one specific technology 
(multimedia) and compare its effectiveness with 
both classroom and text-based instruction in 
teaching project management modules using the 
same teaching materials. The characteristics of the 
instructional material are briefly described to 
provide support to the comparability statement. 

The Textbooks 

The textbooks from which the multimedia CD- 
ROM, and the in-class instmction, are based are two 
major project management publications. 

Frame (1994) The New Project 
Management: Tools for an Age of Rapid 
Change, Corporate Reengineering, and 
Other Business Realities. The Jossey-Bass 
Management Series 

Frame (1995) Managing Projects in 
Organizations: How to Make the Best Use 
of Time, Techniques, and People. The 
Jossey-Bass Management Series 

Both publications are ranked highly (number of 
sales, and reviewers’ comments) in the project 
management literature. They both present similar 
design and layout features, and they both address 
complementary topics. They use diagrams and 



drawings to reinforce understanding, offer several 
examples, and occasionally include mini-case 
studiesto foster reflection and application. The case 
studies present problems, and solutions, to emulate 
a mechanism of feedback provision. 

The multimedia CD-ROM 

The ‘Project Management in Organizations 
(PMO) CD-ROM’ is the multimedia version of 
Frame, 1994 & Frame, 1995. The CD-ROM 
closely follows the books’ organization: 
learning units mirro r chapter titles and structure. 
The application is designed to teach the 
fundamentals of project management. 
Audiences vary from aspiringproject managers, 
project team members, executives, or other 
stakeholders interested in learning project 
management. 

The CD-ROM provides a stand-alone tool for 
learningprojectmanagement introductory topics 
and is copyrighted by the Education Services 
Institute - ESI-International, 1997. It includes 
different media formats and features: 

full motion video 

audio files 

text-based animation 

extensive interactivity (with case 

studies and other exercises) 

high-quality graphics 

user-friendly navigation interface 

It also offers a Toolbox with examples of tools, 
access to Resources (calculator, glossary, 
notepad), and an Organizer, to sort information 
in the program, and access testing. 

In-class instruction 

The lecture presentationsby J. DavidsonFrame 
are supported by the use of overhead 
transparencies. The transparencies follow the 
organization of Frame, 1994, 1995. They 
display the graphical images of the textbook, 
and develop additional graphical 
representations. The instructor primarily uses 
markers to write on transparencies during the 
presentation. Occasionally, display boards are 
used to support explanations. 
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The instructor offers several examples, asks 
frequent questions to the audience, provides 
feedback to responses, and encourages 
participation through in-class discussion of 
section problems. Frequently, cases and other 
exercises are completed in-class, with the 
instructor support. 

3. The study does not address the issue of “novelty. 
Clark’s point b) suggests that the use of technology 
has an initial positive impact on learning, but this 
impact is lowered over time (what Clark calls 
^^novelty effect). Clark himself (Clark & Sugrue, 
1995) reports that some studies on the use of 
computers in college (Kulik & Kulik, 1980) did not 
find any evidence for the novelty effect. Kulik and 
Kulik’s comparisons of studies of one or two hours 
duration, vs. studies whichheld weekly sessionsfor 
an entire semester, reported that the effect sizes on 
learning were roughly the same. Clark responds to 
these findings by pointing to the fact that the use of 
computers may be a less “novel experience for 
college students than for other younger students 
(p.354). 

Because the subjects of this research are college 
students, following Kuliks’ findings and Clark’s own 
considerations, this study limits the length of the 
instructional modules to short-segments of instruction 
(from 1 hour and 15 minutes to 2 hours of instruction). 
Moreover, the limited length ofthe instructionalmodules 
enables focusing on the impact of the medium on 
achievement and satisfaction rate by eliminating 
competing variables (such as “total study time ) that 
studies with delayed post-testing may introduce. The 
researchers recognize that the use of short-instructional 
modules introduces some limitations regarding the types 
of knowle dge and p erforma nee tasks th at are achiev able 
within the limited timeframe. 

METHODOLOGY AND FOCUS 
REFINEMENT THROUGH PILOT STUDIES 

Three pilot studies precede the refinement of both the 
theoretical framework and the procedures described in 
this study. They are valuable elements of the quasi- 
experimental approach and are, therefore, reported in 
this paper. 



Research Methodology Refinement - Pilot Study (1) 

A first pilot study has been instrumental to refine the 
methodological approach (and decidingbetween a short- 
term approach vs. a longer-term study). The researchers’ 
aim to exploit the advantages of the media by offering 
participants the opportunity to use the textbook/s, or the 
CD-ROM at their own timing, outside of a laboratory 
environment, and with little control of the instructional 
conditions has been prevented by a high-mortality rate. 

The sample population for this first pilot designed to 
assess the feasibility of a long-term self-paced study was 
composed of working professionals registered to 
monthly workshops in Project Management offered by 
a major project management educational services 
institution in the US. Individuals registered to the 
workshop received an invitation to participate in the 
study. Volunteers were randomly assigned to two 
treatments (interactive multimedia CD-ROM or 
textbook-instruction). Individuals registered in the 
workshop were assigned to the lecture treatment. They 
completed paper-and-pencil surveys. Volunteers in the 
other treatment groups took all the surveys on-line. 

The outcomes of this pilot study supported the need of 
controlling for instructional conditions, thus undertaking, 
the study in a real-laboratory setting. Although 
interactive multimedia applications are designed to 
support self-paced instruction, learners need to receive 
constant reinforcement and feedback to complete the 
training program. Although over 23% of the 1 00- 
individuals contacted volunteered their participation, 
13% dropped prior to receiving the instructional 
materials. Of the remaining 10% who received the 
instructional materials, only 5 people completed the 
study on time (prior to their scheduled training session). 
The high mortality “ rate has been instrumental in 
redesigning the research to test, only in a controlled 
setting, graduate and undergraduate groups enrolled in 
business administration degree programs. 

Demographic Sample Pilot Study (2) 

Another pilot study was conducted with the objective of 
testingseale reliability and assesses the comparability of 
MBA-Cohort groups. The pilot was held during the 
laboratory sessions of two sections of an MBA-Cohort 
Information System course. Students enrolled in 
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Tuesday and Wednesday’s laboratory sessions (hereafter 
referred to as Tuesday, or Wednesday students) took an 
on-line questionnaire on project management topics 
ranging from Time Management, Cost Management, 
Human Resources Management, Risk Management, 
Communication Managemen t, and Scope Management. 
They also completed a demographic survey. 

The purpose of the demographic survey was to compare 
the characteristics of the Tuesday and Wednesday 
groups to identify any specific group composition bias, 
which could inhibitthe generalizability of the findings, 
and caution against the use of the scale in the actual 
study. The demographic data evaluated group variances. 
The frequency results of the demographic surveys 



confirmed that difference between the two groups 
(Tuesday and Wednesday) in this population are 
minimal: 

65.5% of the Tuesday participants are Asians/Pacific 
Islanders, 64.5% of the students in the Wednesday 
group report belonging to the same ethnicity group. 
18.8% (T) vs. 26.9% (W) are CaucasiansAVhite; 
75% (T) vs. 81.5% (W) have a college degree, 
21.9% (T) vs. 14.8% (W) have a masters degree; 
15.6% (T) vs. 14.8% (W) work in a management 
field, 68.8% (T) vs. 70.4% (W) are in the education 
field (mostly as students); - .yee Figure 2 - 
Additionally: 96.9% (T) vs. 92.6% (W) are between 
20 and 29 years old - See Figure 3 - 



FIGURE 2 

PRIOR WORK EXPERIENCE 
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FIGURE 3 

AGE OF THE LEARNERS 



Wednesday Group Demographics 



ACE 





Frequency 


Percent 


V^lid 

Percent 


Cumulative 

Percent 


V^lid 20-29 


31 


96.9 


96.9 


96.9 


30-39 


1 


3.1 


3.1 


100.0 


Total 


32 


100.0 


100.0 




Total 


32 


100.0 







Tuesday Group Demographics 
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43.8% (T) vs. 55.6% (W) report of having been using a computer for 7 years or more; 

50% (T) vs. 59.3% (W) are male; 

37.5% (T) vs. 40.7% (W) report of having used the computer primarily at work/school, but also at home. 



The frequencies values reported above display ranges of 
differences from 0.9% to less than 10% difference on 
same measures, with the mode of occurrences 
concentrating on the lower differences. Because more 
recent assignments of students to cohort groups have 
standardized stratification mechanisms, it is expected 
that similarities in the groups are further increased in the 
groups participating in the redesigned study. 

Satisfaction & Attitudes - Pilot Study (3) 

A pilot test of the undergraduate students (senior level) 
was undertaken in one section of a ‘Change in 
Organizations’ course. The main objective of the pilot 
was to collect feedback on undergraduate students’ 
appreciation of the selected project management topic, 
and their attitudes toward the experimental procedures 
(i.e. special attention focuses on their reactions to the 
presentations, and their interest in attending another 
session on the topic). Because the sample size of the 
pilot group was small (44 students present in the 
sessions), and their subgroups were composed of only 1 5 



or less students, the results from the statistical analysis 
are not reliable, and are omitted in this discussion. 
Interesting findings, however, emerge from the 
descriptive frequencies analysis of satisfaction, and 
appeal measures (see Figure 5 and Figure 6). 

Figure 4 shows that students in the lecture group (#1) 
were the most satisfied with the organization, pace, 
content of instruction, and with the instructor’s 
presentation (means > than 4.5). The other groups’ 
satisfaction was also above the mid-point (mean = 4), 
with the CD-ROM group being more satisfied than the 
textbook group with the organization of the materials 
(ORGANIZ). Atthe same time, the CD-ROM group was 
the least satisfied in terms of content, pace, and 
presentation (FRAME). Although the size of each 
difference was within a 1 -point difference, the CD-ROM 
participants were consistently least satisfied with the 
materials, and felt that they did not learn the material 
satisfactorily (LEARN). 
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FIGURE 4 

SATISFACTION SURVEY DATA, 
UNDERGRADUATE PILOT STUDY, FALL 1999 
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It is interesting to compare the values in Figure4 with 
other results. Students in the CD-ROM group (#2) 
reported that they had the least prior knowledge of the 
topic (Figure 5a), but they obtained the highest pretest 
and posttest mean scores (Figure 5b), with the 
limitations that the sample size is not sufficient to use 
formal parametric procedures, as well as draw 
conclusions from comparative differences. 

FIGURE 5 

PRIOR KNOWLEDGE OF THE TOPIC 
Figure 5a: Self-reported Prior Knowledge 
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Additionally, 30% of the students answered that they 
would like to attend another instmctional session similar 
to the one in which they participated; and 4 1 % answered 
thatthey ‘may’ prefer attending a similar session (Figure 
6 ). 



Figure 5b: Pre-TEST — Posttest Differences 

Pretest-Posttest by Groups 
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FIGURE 6 

APPEAL RESPONSE, UNDERGRADUATE 
PILOT STUDY, FALL 1999 
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The appeal measure is encouraging, especially if 
contrasted with the conditions of the pilot study. 
Students in this pilot were substantially rushed into the 
instructional materials because of the logistic 
arrangements, the lack of proximity of the laboratory and 
the reading facilities, the need to perform other 
miscellaneous administrative tasks, substantially 
reducinginstructionaltime. TheCD-ROMgroup did not 
comp lete all the assigned sessions, and spent only a total 
of 17-minutes using the application. Similar time- 
constraints characterized the textbook-group; whereas 
the in-class group is the only group that used the time 
satisfactorily. 

The conclusions of this pilot (#3) led to an improvement 
in scheduling efforts, with each group meeting directly 
in the classroom in which the treatment is offered, and 
reassured the researcher of the appropriateness of using 
the ‘change control’ topic with this group of students. 
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Lesson learned from this, and the other pilots, guided the 
implementation of better-enforced laboratory 
procedures. 

CONCLUSIONS FROM THE PILOT STUDIES 

The implementation of the pilot studies has been critical 
to the design of an improved methodology. First of all, 
it has demonstrated that the ambition to conduct such 
experiments outside controlled lab environments is 
tainted with the inability to controlthe learningprocess. 
It results in high-mortality due to conflicting tasks and, 
above all, lack of reliability of self-reported indication 
on how/for-how-long the CD-ROM was used. The 
evaluation of the comparative effectiveness can be 
conducted only if both the instructional material and the 
instructional conditions are equivalent (controlled 
laboratory setting). This c alls for the application of these 
types of experiments only fora limited amount of time, 
unless the researcher has full control on the many 
intervening variables (study and practice time, etc.). It is 
no surprise that similar experiments have led to conclude 
that there are “no differences in learning across 
instructional materials . They did not take into account 
the study-time/efforts of the learners, which obviously 
influences success in the final exam regardless of the 
tool used to study the instructional material. 

Secondly, the pilot studies have helped assess the 
comparability of the groups being tested and refined the 
survey instruments. Thirdly, they havehelped screen out 
learners’ attitude (both graduate and undergraduate 
students) towards the study material, which is also 
another important component of students’ learning. 

VALUE OF THE STUDY 

Once if is completed and implemented with the changes 
suggested by the pilot studies herein described, this 
study will contribute to research on instructional 
technology effectiveness in several ways. 

It will offer a unique comparative situation, with 
coincidenttopic presentation across media (multimedia, 
textbook, in-class instruction). Therefore, it will enable 
the researcher to control for content-differences biases 
found in earlier studies. The research design overcomes 
several limitations of these earlier studies (controlling 
for time, instructor, and content-differences), and 
presents a framework that addresses previous concerns. 



This research will contribute to the advancement of the 
debate on effectiveness of instructional technologies by 
looking at one specific medium, and its comparative 
impact. In fact, it responds to calls for further research in 
areas that have key instructional implications for 
education and training. 

By replicating this research with undergraduate and 
graduate students, with higher and lower complexity 
topics, and contrasting direction and magnitude (effect 
size), this research offers further validation procedures. 
The replication also differentiates learning and 
satisfaction by groups (undergraduate and graduate), and 
topic complexity. Statistically significant differences 
among the age groups, for example, allow the 
measurement ofmedia effectiveness on differentlearner 
populations. Learning goals may not be achieved 
because the learner’s developmental stage (Piaget & 
Inhelder, 1969) has not reached the level of 
sophistication that benefits from the medium 
instructional approach. Or, they may not be achieved 
because the complexity of the topic requires different 
instructional representationsto fosterunderstandingfirst, 
then, retention and application. This study helps 
investigate these relationships by including these 
dimensions in the research design. 
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